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Abstract 


Verbal autopsies (VA) are widely used to provide cause-specific mortality estimates in de¬ 
veloping world settings where vital registration does not function well. VAs assign cause(s) 
to a death by using information describing the events leading up to the death, provided by 
care givers. Typically physicians read VA interviews and assign causes using their expert 
knowledge. Physician coding is often slow, and individual physicians bring bias to the coding 
process that results in non-comparable cause assignments. These problems significantly limit 
the utility of physician-coded VAs. A solution to both is to use an algorithmic approach that 
formalizes the cause-assignment process. This ensures that assigned causes are comparable 
and requires many fewer person-hours so that cause assignment can be conducted quickly 
without disrupting the normal work of physicians. Peter Byass’ Inter VA method (Byass 


et ah, 2012) is the most widely used algorithmic approach to VA coding and is aligned with 


the WHO 2012 standard VA questionnaire (Leitao et ah, 2013) 


The statistical model underpinning InterVA can be improved; uncertainty needs to be quan¬ 
tified, and the link between the population-level CSMFs and the individual-level cause as¬ 
signments needs to be statistically rigorous. Addressing these theoretical concerns provides 
an opportunity to create new software using modern languages that can run on multiple plat¬ 
forms and will be widely shared. Building on the overall framework pioneered by InterVA, 
our work creates a statistical model for automated VA cause assignment. 
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1 Introduction 


Verbal autopsy (VA) is a common approach for determining cause of death in regions 
where deaths are not recorded routinely. VAs are a standardized questionnaire adminis¬ 
tered to caregivers, family members, or others knowledgeable of the circumstances of a 


recent death with the goal of using these data to infer the likely causes of death (Byass 


et ah, 2012). The VA survey instrument asks questions related to the deceased individ¬ 


ual’s medical condition (did the person have diarrhea, for example) and related to other 
factors surrounding the death (did the person die in an automotive accident, for exam¬ 
ple). VA has been widely used by researchers in Health and Demographic Surveillance 


Sites (HDSS), such as the INDEPTH Network (Sankoh and Byass, 2012), and has recently 


received renewed attention from the World Health Organization through the release of an up¬ 
date to the widely used standard VA questionnaire (see http: //www. who. int/healthinf o/ 
statistics/verbalautopsystandards/). The main statistical challenge with VA data is 
to ascertain patterns in responses that correspond to a pre-dehned set of causes of death. 
Typically the nature of such patterns is not known a priori and measurements are subject 
to various types of measurement error, which are discussed in further detail below. 

There are two credible methods for automating the assignment of cause of death from VA 
Inter VA (see for examr 


data. Inter VA (see for example: Byass et ah, 2003, 2006, 2012) is a proprietary algorithm 
developed and maintained by Peter Byass, and it has been used extensively by both the AL¬ 


PHA (Maher et al., 2010) and INDEPTH networks of HDSS sites and a wide variety of other 
research groups. At this time Inter VA appears to be the de facto standard method. The 
Institute for Health Metrics and Evaluation (IHME) have proposed a number of additional 


methods (for example: 

Flaxman et al. 

2011[ James et al. 

2011 

Murray et al.[ 2011 

), some of 

which build on earlier work by King and Lu (King et al. 

2010 

King and Lu, [2008 

). Among 


their methods, the Simplihed Symptom Pattern Recognition (SSPR) method (Murray et al., 
2011) is most directly comparable to InterVA and appears to have the best chance of becom¬ 
ing widely used. The SSPR and related methods require a so-called ‘gold standard’ database, 
a database consisting of a large number of deaths where the cause has been certihed by medi¬ 
cal professionals and is considered reliable, and further, where the symptoms for those deaths 
are also verihable by medical professionals. Deaths recorded in a gold standard database 
are typically in-hospital deaths. Given regional variation in the prevalence and familiarity of 
medical professionals with certain causes of death, region-specihc gold standard databases 
are also necessary. The majority of public health and epidemiology researchers and officials 
do not have access to such a gold standard database, motivating development of methods to 
infer cause of death using VA data without access to gold standard databases. 


Aside from the challenges related to obtaining useful gold standard training data, using VA 
data to infer cause assignment is also statistically challenging because there are multiple 
sources of variation and error present in VA data. We identify three sources of variation 
and propose a novel method called InSilicoVA to address these sources of variation. First, 
though responses on the VA instrument may be the respondents recollection or best guess 
about what has happened, they are not necessarily accurate. This type of error is present 
in all survey data but may be especially pronounced in VA data because respondents have 
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varying levels of familiarity with the circumstances surrounding a death. Also, the dehnition 
of an event of interest may be different for each respondent. A question about diarrhea, 
for example, requires the respondent to be sufficiently involved in the decedent’s medical 
care to know this information and to have a dehnition of diarrhea that maps relatively 
well to accepted clinical standards. A second source of variability arises from individual 
variation in presentation of diseases. Statistical methods must be sufficiently robust to 
overhtting to appreciate that two individuals with the same cause of death may have slightly 
different presentations, and thus, different VA reports. Third, like InterVA, InSilicoVA 
will use physician elicited conditional probabilities. These probabilities, representing the 
consensus estimate of the likelihood that a person will experience a given symptom if they 
died from a certain cause, are unknown. Simulation and results with data from the Agincourt 
MRC/Wits Rural Public Health and Heath Transitions Research Unit (Agincourt) indicate 
that results obtained with conditional probabilities assumed hxed and known (as is done in 
both InterVA and the SSPR method) can underestimate uncertainty in population cause of 
death distributions. We evaluate the sensitivity to these prior probabilities. 

InSilicoVA incorporates uncertainty from these sources and propagates the uncertainty be¬ 
tween individual cause assignments and population distributions. Accounting for these 
sources of error produces conhdence intervals for both individual cause assignments and 
population distributions. These conhdence intervals rehect more realistically the complex¬ 
ity of cause assignment with VA data. A unihed framework for both individual cause and 
population distribution estimation also means that additional information about individual 
causes, such as physician coded VAs, can easily be used in InSilicoVA, even if physician coded 
records are only available for a subset of cases. Further, we can exactly quantify the contri¬ 
bution of each VA questionnaire item in classifying a case into a specihc cause, affording the 
possibility for ‘item reduction’ by identifying which inputs are most useful for discriminating 
between causes. This feature could lead to a more efficient, streamlined survey mechanism 
and set of conditional probabilities for elicitation. 

In Section we describe InterVA and present open challenges which we address with InSil¬ 
icoVA. In Section we present the InSilicoVA model. Section describes applications of 
both methods to simulated and real data, and Section [473] provides the results. We conclude 
with a very brief discussion in Section 


2 InterVA 

2.1 Byass’ Description of InterVA 


Below is Byass’ summary of the InterVA method presented in 


Bayes theorem links the probability of an event happening given a particular circumstance 
with the unconditional probability of the same event and the conditional probability of the 
circumstance given the event. If the event of interest is a particular cause of death, and the 
circumstance is part of the events leading to death, then Bayes theorem can be applied in terms 
of circumstances and causes of death. Specihcally, if there are a predetermined set of possible 


Byass et ah (2012 
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causes of death Ci... Cm and another set of indicators /i... representing various signs, 
symptoms and circumstances leading to death, then Bayes general theorem for any particular 
Ci and Ij can be stated as: 


P(r\j^= P(/,-|c.)xP(Q) 

^ P{Ij\Ci) X P{Ci) + P{Ij\\Ci) X P{\Ci) 


( 1 ) 


where P(!C'i) is (1— P(C'i)). Over the whole set of causes of death Ci... Cm a set of probabilities 
for each Ci can be calculated using a normalising assumption so that the total conditional 
probability over all causes totals unity: 


pmio) 


P{I,\Ci) X P{Ci) 

Etim) 


( 2 ) 


Using an initial set of unconditional probabilities for causes of death Ci... Cm (which can 
be thought of as P(C'i|/o)) and a matrix of conditional probabilities P{Ij\Ci) for indicators 
Ii.. .In and causes Ci... Cm, it is possible to repeatedly apply the same calculation process 
for each /i... that applies to a particular death: 


P(Q|/i..,„) 


P(/,|C',) xP(C',|/o...„-i) 


(3) 


This process typically results in the probabilities of most causes reducing, while a few likely 
causes are characterised by their increasing probabilities as successive indicators are processed. 


In the same article Byass describes the process of dehning the conditional probabilities 
P(/,|Q). 


Apart from the mathematics, the major challenge in building a probabilistic model covering 
all causes of death to a reasonable level of detail lies in populating the matrix of conditional 
probabilities P{Ij\Ci). There is no overall source of data available which systematically quan¬ 
tifies probabilities of various signs, symptoms and circumstances leading to death in terms of 
their associations with particular causes. Therefore, these conditional probabilities have to be 
estimated from a diversity of incomplete sources (including previous InterVA models) and mod¬ 
ulated by expert opinion. In the various versions of InterVA that have been developed, expert 
panels have been convened to capture clinical expertise on the relationships between indicators 
and causes. In this case, an expert panel convened in Geneva in December 2011 and contin¬ 
ued to deliberate subsequently, particularly considering issues that built on previous InterVA 
versions. Experience has shown that gradations in levels of perceived probabilities correspond 
more to a logarithmic than linear scale, and in the expert consultation for InterVA-4, we used a 
perceived probability scale that was subsequently converted to numbers on a logarithmic scale 
as shown below. 
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Table 1: InterVA Conditional 

Probability Letter-Value 
Correspondances 


Label 

Value 

Interpretation 

I 

1.0 

Always 

A 

0.8 

Almost always 

A 

0.5 

Common 

A 

0.2 


B 

0.1 

Often 

B 

0.05 


B 

0.02 


C 

0.01 

Unusual 

C 

0.005 


C 

0.002 


D 

0.001 

vRare 

D 

0.0005 


D 

0.0001 


E 

0.00001 

Hardly ever 

N 

0.0 

Never 


The physician-derived conditional probabilities that are snpplied with the Inter VA software 
are coded nsing the letter codes in the leftmost colnmn of Table 

We rewrite, interpret and discnss the InterVA model below. 


(Byass, 2013 


2.2 Our Notation for InterVA: 

• Deaths: yj j e {1,..., J}, V = [t/i,..., yj] 

• Signs/symptoms: Sk G {0,1}, A: G {1,..., A'}, S = [si,..., Sx] 

• Canses: c„ n G {1,..., N], Yl!l=i Cj,n = 1 

• Fraction of all deaths that are canse n, the ‘canse-specihc mortality fraction’ (CSMF): 

A nG{l,...,iV}, Eli/n = l 

2.3 Our Description of InterVA Data Requirements: 

1. For each death y^, a VA interview prodnces a binary-valned vector of signs/symptoms: 

Sj (4) 

S is the J X K matrix whose rows are the Sj for each death. 

2. A K X N matrix of conditional probabilities reflecting physicians’ opinions abont how 
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likely a given sign/symptom is for a death resulting from a given cause: 


Pr(si|ci) Pr(si|c 2 ) ■■■ Pr(si|cAr) 
Pr(s2|ci) Pr(s2|c2) ■■■ Pr(s2|cAr) 


Pr(sx|ci) Pr(sK|c2) ■■■ Pt{sk\cn) 


As supplied with the InterVA software (Byass, 2013) P does not contain internally con¬ 
sistent probabilities This is easy to understand by noting that these probabilities are 
not derived from a well-dehned event space that would constrain them to be consistent 
with one another. As described by Byass above in Section 2T the physicians provide 
a ‘letter grade’ for each conditional probability, and these correspond to a ranking 
of perceived likelihood of a given sign/symptom if the death is due to a given cause. 
These letter grades are then turned into numbers in the range [0,1] (NB: 0.0 and 1.0 
are included) using Table 


Consequently it is not possible to assume that the members of P will behave as expected 
when one attempts to calculate complements and use more than one in an expression 
in a way that should be consistent. 

3. An initial guess of F, F' = f'p^] 


2.4 Our Presentation of the InterVA Model and Algorithm 


For a specihc death yj we can imagine and examine the two-dimensional joint distribution 


where 


I ^j,k) 




a) ■ Pr(Cj^n) 

Pri 




^n) ■ Pr(Cj,n) + X 


X = Pr(sj-fc|-'Cj>) ■ Pr(-.Cj>) 


( 6 ) 

(7) 


Looking on the RHS of we have PT{sj^k\cj,n) from the conditional probabilities from 
physicians and Pr(cj_n) ~ /„• h fhe conditional probabilities P were well-behaved, then 


N 


X = 


^ ^ Pr(Sj^fc|Cj^ri,') ■ 


( 8 ) 


n'=l, n'^n 


^ The P supplied with InterVA has many logical inconsistencies, for example situations where con¬ 
ditional probabilities should add up to equal another: Pr(fast breathing for 2 weeks or longer | HIV) -I- 
Pr(fast breathing for less than 2 weeks | HIV) ^ Pr(fast breathing | HIV), or where they just do not make 
sense: Pr(fast breathing for 2 weeks or longer | sepsis) > Pr(fast breathing | sepsis). The P supplied with 
InterVA-4 (Byass 20131 is a 254 x 69 matrix with 17,526 entries. We have investigated automated ways 


of correcting the inconsistencies, but with every attempt we discover more, so we have concluded that the 
entries in P need to be re-elicited from physicians using an approach that ensures that they are consistent. 
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However since the P supplied with the InterVA software (Byass, 2013) are not consistent 
with one another this calculation does not produce useful results. 


InterVA solves this with an arbitrary reformulation of the relationship. For each death yj 
and over all signs/symptoms Sk associated with yf 


K 

V (j,n) : Propensity(cj>|4) = fn Yl 

k=l 


(9) 


For each death yj these ‘Propensities’ do not add to 1.0 so they need to be normalized to 
produce well-behaved probabilities: 


w .. s ^ f . Propensity(cj'„|5j) 
V (j, n) : Pr(c^-„) = ^^ 


En=i Propensity(cj- 


( 10 ) 


The population-level CSMFs F are calculated by adding up the results of calculating (10) 
for all causes for all deaths: 


V n : fn = '^ Pr(cj-^ 
i=i 


( 11 ) 


2.5 Evaluation of InterVA Model & Algorithm 

In effect what InterVA does is distribute a given death among a number of predehned causes. 
The cause with the largest fraction is assumed to be the primary cause, followed with decreas¬ 
ing signihcance by the remaining causes in order from largest to smallest. The conceptual 
construct of a ‘partial’ death is central to InterVA and is interchanged freely with the prob¬ 
ability of dying from a given cause. This equivalence is not real and is at the heart of the 
theoretical problems with InterVA. 

At a high level InterVA proposes a very useful solution to the fundamental challenge that 
all automated VA coding algorithms face - how to characterize and use the relationship 
that exists between signs/symptoms and causes of death. In a perfect world we would have 
medically certihed patient records that include the results of ‘real’ autopsies, and we could 
use those to model this relationship and use the results of that model in our cause-assignment 
algorithms. But in that perfect world there is no use for VA at all. So by dehnition we live 
in the world where that type of ‘gold standard’ data do not and will not exist most of the 
time for most of the developing world where VAs are conducted. Byass’ solution to this is 
to accept the limitations on the expert knowledge that physicians can provide to substitute 
for gold standard data, and further, to elicit and then organize that information in a very 
useful format ~ the conditional probabilities matrix P above in (|^. In sum. Byass has sorted 
through a variety of possible general strategies and settled on one that is both doable and 
produces useful results. 

Where we contribute is to help rehne the statistical and computational methods used to 
conduct the cause assignments. In order to do that we have evaluated InterVA thoroughly. 
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and we have identified a nnmber of weaknesses that we feel can be addressed. The brief 
list below is by necessity a blnnt description of those weaknesses, which nevertheless do not 
rednce the importance of InterVA as described jnst above. 


There are several theoretical problems with the InterVA model; 


1. The derivation presented in Q throngh is incorrect; in particnlar ( [2| d oes not follow 
from Q, and ([^ does not follow from Q. As described jnst above, (11) reqnires that 
probabilities be eqnated with fractional deaths, which is conceptnally difficnlt. 


2. InterVA’s statistical model is not ‘probabilistic’ in the recognized sense becanse it 
does not inclnde elements that can vary nnpredictably, and hence there is no random¬ 
ness. Althongh P contains ‘probabilities’ (see discnssion with ([^ above), these are 
not allowed to vary in the estimation procednre nsed to assign canses of death - P is 
effectively a fixed inpnt to the model. 


3. Becanse the model does not contain featnres that are allowed to vary nnpredictably, it 
is not possible to qnantify nncertainty to prodnce probabilistic error bonnds. 

4. If we ignore the errors in the derivation and work with ([^ - ( [TT| ) as if they were correct, 
there are additional problems. Eqnation ([^ is at the core of InterVA and demonstrates 
two nndesirable featnres: 


(a) For a specific individnal the propensity for each canse is deterministically affected 
by f^, what Byass terms the ‘prior’ probability of canse n, effectively a nser- 
defined parameter of the algorithm. This means that the final individnal-level 
canse assignments are a deterministic transformation of the so-called ‘prior’ - i.e. 
the resnlts are not only sensitive to bnt depend directly on the ‘prior’. 

(b) The expression in (§ captnres only one valance in the relationship between 
signs/symptoms and a canse of death - it acknowledges and reacts only to the 
presence of a sign/symptom bnt not to its absence, effectively throwing away half 
of the information in the dataset. To inclnde information conveyed by the absence 
of a sign/symptom, 0 needs a term that involves something like ‘1 —Pr(sj,A:|Cn)’- 
[The components of InSilicoVA in ( [2^ and ( [2^ below inclnde this term.] This is 
nndesirable for two reasons: (1) signs/symptoms are not selecting canses that fit 
and de-selecting canses that don’t, bnt raiher just de-selecting canses, and (2) the 
final probabilities are typically the prodnct of a large nnmber of very small nnm- 
bers, and hence their nnmeric valnes can become extremely small, small enongh to 
interact badly and nnpredictable with the nnmerical storage/arithmetic capacity 
of the software and compnters nsed to calcnlate them. A simple log transforma¬ 
tion wonld solve this problem. 

5. Finally, ( pT| is a deterministic transformation of the individnal-level canse assignments 
to prodnce a popnlation-level CSMF; simply another way of stating the individnal-level 
canse assignments. Because the individual-level cause assignments are not probabilis¬ 
tic, neither are the resulting CSMFs. 


In addition there are idiosyncrasies that affect the current implementation of InterVA (Byass 
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2013) and some oddities having to do with the matrix of conditional probabilities provided 


with Byass’ InterVA. We will not describe those here. 


InSilicoVA is designed to overcome these problems and provide a valid statistical framework 
on which further rehnements can be built. 


3 InSilicoVA 

InSilicoVA is a statistical model and computational algorithm to automate assignment of 
cause of death from data obtained by VA interviews. Broadly the method aims to: 

• Follow in the footsteps of InterVA building on its strengths and addressing its weak¬ 
nesses. 

• Produce consistent, comparable cause assignments and CSMFs. 

• Be statistically and computationally valid and extensible. 

• Provide a means to quantify uncertainty. 

• Be able to function to assign causes to a single death. 

The name ‘InSilicoVA’ is inspired by ‘in-vitro’ studies that mimic real biology but in more 
controlled circumstances (often on ‘glass’ petri dishes). In this case we are assigning causes 
to deaths using a computer that performs the required calculations using a silicon chip. 
Further, we owe a great debt to the InterVA (interpret VA) method that provides useful 
philosophical and practical frameworks on which to build the new method - so we have stuck 
to the structure of InterVA’s name. 


3.1 InSilicoVA Notation: 

• Deaths: yj j G {1,..., J}, V = [|/i,..., 2 /j] 

• Signs/symptoms: G {0,1}, /c G {1,..., iF}, S = [si,..., s^] 

• Causes of death: c„ n G {1,..., V}, C = [ci,..., Cn] 

• For individual j, probability of cause n given Sj: ij^n = Pr(j/j = Cn\Sj), j G {1,..., J}, 
n G {1, . . . , V}, Lj = [/j- 1 , . . . , /j'Tv], Yln=l = 1 

• Count of all deaths that are cause n, the ‘cause-specihc death count’ (CSDC): 
rrin n G {1,..., V}, M = [rrii,.. .,171^], Y.n=i'^n = J 

• Fraction of all deaths that are cause n, the ‘cause-specihc mortality fraction’ (CSMF): 
A nG {!,..., AT}, F=[/i,...,/^], Eli/n = l 



3.2 


InSilicoVA Data: 


1. For each death yj, the VA interview produces a binary-valued vector of signs/symptoms: 


5 • ■ • ^j,K} 


( 12 ) 


S is the J X K matrix whose rows are the Sj for each death. 


The columns of S are 
assumed to be independent given (F, i.e. there is no systematic relationship between the 
signs/symptoms for a given cause. This is very obviously not a justihable assumption. 
Signs and symptoms come in characteristic sets depending on the cause of death, so 
there is some correlation between them, conditional on a given cause. Nonetheless we 
assume independence in order to facilitate initial construction and testing of our model, 
and most pragmatically, so that we can utilize the matrix of conditional probabilities 
supplied by Byass with the Inter VA software (Byass, 2013) - it is impossible to either 


regenerate or signihcantly improve upon these without signihcant resources with which 
to organize meetings of physicians with the relevant experience who can provide this 
information. This Sj is the same as 0 used by InterVA. 

2. A K X N matrix of conditional probabilities reflecting physicians’ opinions about how 
likely a given sign/symptom is for a death resulting from a given cause: 


Pr(si|ci) Pr(si|c 2 ) 
Pr(s2|ci) Pr(s2|c2) 

Pr(sx|ci) Pr(s;^|c2) 


Pr(si|cAr) 

Pr(s2|cv) 

Pr(Si^|cAr) 


(13) 


InSilicoVA assumes that the components of P are consistent with one another. In the 
simulation study described in Section |4.1| , we construct consistent values for P, but 
when we test the model on real data in Section [T^ we have no option other than using 


the inconsistent P supplied with the InterVA software (Byass, 2013). 

3. An initial guess of F, F' = /)^] 


3.3 InSilicoVA Algorithm 

We are interested in the joint distribution [F, Y) given the set of observed signs/symptoms 
S. The posterior distribution is: 


rr(Fy|S) Pr(S|y,J?)Pr(F|F)Pr(F) 


oc Pr(S|V,F)Pr(V|F)Pr(F) 

(14) 

= nP^(S|%',^)Pr(%|F)Pr(F) 
i=i 

(15) 
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Because individual cause assignments are independent, individual sign/symptom vectors Sj 
are independent from F (the CSMFs), and we have: 

j 

Pr(F,F|S) oc JJPr(S| 2 /j)Pr( 2 /j|F)Pr(F) (16) 

i=i 

We will use a Gibbs sampler to sample from this posterior as follows: 

G.l start with an initial guess of F, F' 

G.2 sample Y\F, S 
G.3 sample F\Y, S 

G.4 repeat steps G.2 and G.3 until F and Y converge 

This algorithm is generic and allows a rich range of models. For the moment the InsilicoVA 
model is: 


^jik 1 

~ Bernoulli(Pr(sj-fc|c„)) 

(17) 

= Cn\F 

~ Multinomial 7 v(l, A) 

(18) 

F 

~ Dirichlet(a), a is Wdimensional and constant 

(19) 


Then the posterior in (16) is: 


J K 


Pr(F,F|S) cx nn Fiisj^klUj = Cn) Pr(|/j = c„|F) Pr(F) 

j=l k=l 


( 20 ) 


This formulation is computationally efficient because of Multinomial/Dirichlet conjugacy, 
and because using Bayes rule we have, for step G.2: 

i/j = Cn\F, Sj ~ MultinomialAr(l, Lj) (21) 


where the ij^n that compose Lj are: 




Pr(%' = Cn\Sj) 

Fijyj = Cn) • Fr{Sj\yj = c„) 

Pr(^,) 

substituting /„ = Pr(?/j = c„) and using the data Sj and P to calculate 
the probability of a specihc Sj given the cause assignment yj = c„ 


fn ■ nf=l (Pr(syfc||/j = ■ [1 - FliSj^klVj = Cn)f^ 

fn' ■ nf=l (Pr(Si,fc|l/j = ■ [1 - Pr(Si,fc|l/i = 


( 22 ) 
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We can also derive from (20) the distribution of F conditional on F, for step G.3: 

F|F, S ~ Dirichlet(M + a) 


( 23 ) 


where 


j 

THn = E [Vj = Cn] , using Iverson’s bracket notation: [z] 


1 li z true 
Q ii z false 


(24) 


In summary, the Gibbs sampler proceeds given suitable initialization F' by: 

G.2 sampling a cause for each death to generate a new Y\F^ S: 

Vj C-n\F 1 Sj rNj MultinomialAr(l, Lj) 

where 

/» ' nhi = Cn)‘‘* ■ |1 - Pr(Si,tl9j = c„)]*'“*'-‘’() 

EGi /-' ■ nhi (Pr(si,tl!/j = c„,Yi* ■ |1 - Pr(s,,t|!/j = c,,.)!*'"''*') 


(25) 


(26) 


G.3 sampling a new F\Y , S: 


F|F, S ~ Dirichlet(M + a) 


(27) 


The resulting sample of {F,Y) and the Lj that go with it form the output of the method. 
These are distributions of GSMFs at the population level and probabilities of dying from 
each cause at the individual level. These distributions can be summarized as required to 
produce point values and measures of uncertainty in F and Lj. 


Deaths often result from more than one cause. InSilicoVA accommodates this possibility by 
producing a separate distribution of the probabilities of being assigned to each cause; that 
is N distributions, one for each cause. In contrast, InterVA reports one value for each cause, 
and those values sum to unity across causes for a single death. 


Finally, with a suitable F, InSilicoVA can be used to assign causes (and their associated ij^n) 
to a single death by repeatedly drawing causes using (25). This requires no more information 
than InterVA to accomplish the same objective, and it produces uncertainty bounds around 
the probabilities of being assigned to each cause. 


4 Testing Comparing InSilicoVA and InterVA 


To evaluate both InSilicoVA and InterVA we £t them to simulated and real data. We have 
created R code that implements both methods. The R code for InterVA matches the results 
produced by Peter Byass’ implementation (Byass, 2013). 
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4.1 Simulation Study 


Our simulated data are generated using this procedure: 


1. Draw a set of Pr(sfc|c„) so that they have the same distribution and range as those 
provided with Byass’ InterVA software (Byass, 2013). 


2. Draw a set of simulated deaths from a made up distribution of deaths by cause. 

3. For each simulated death, assign a set of signs/symptoms by applying the conditional 
probabilities simulated in step 1. 


These simulated data have the same overall features as the data required for either InterVA 
or InSilicoVA, and we know both the real population distribution of deaths by cause and the 
true individual cause assignments. 


Our simulation study poses three questions: 


1. Fair comparison of InSilicoVA and InterVA. To make this comparison we gen¬ 
erate 100 simulated datasets and apply both methods to each dataset. We summarize 
the results with individual-level and population-level error measures. We refer to this 
as ‘fair’ because we apply both methods in their simplest form to data that fulhll all 
the requirements of both methods. Since the data are effectively ‘perfect’ we expect 
both methods to perform well. 

2. Influence of numeric values of Pr(sfe|c„). Given the structure of InterVA, we are 
concerned that the results of InterVA may be sensitive to the exact numerical values 
taken by the conditional probabilities supplied by physicians. To test this we rescale 
the simulated Pr(sfc|c„) so that their range is restricted to [0.25 — 0.75]. We again 
simulate 100 datasets and apply both methods to each dataset and summarize the 
resulting errors for each method. 

3. Reporting errors. As we mentioned in the introduction, we are concerned about 
reporting errors for any algorithmic approach to VA cause assignment. To investigate 
this we randomly recode our simulated signs/symptoms so that a small fraction are 
‘wrong’ - i.e. coded 0 when the sign/symptom exists (15%) or 1 when there is no 
sign/symptom (10%). We do this for 100 simulated datasets and summarize errors 
resulting from application of both methods. 


4.2 Application to Real Data - Agincourt HDSS 


To investigate the behavior of InSilicoVA and InterVA on real data, we apply both methods to 
the VA data generated by the Agincourt health and demographic surveillance system (HDSS) 


in South Africa (Kahn et ah, 2012) from roughly 1993 to the present. The Agincourt site 


continuously monitors the population of 21 villages located in the Bushbuckridge District 
of Mpumalanga Province in northeast South Africa. This is a rural population living in 
what was during Apartheid a black ‘homeland’. The Agincourt HDSS was established in the 
early 1990s with the purpose of guiding the reorganization of South Africa’s health system. 


12 








Since then the goals of the HDSS have evolved and now it contributes to evaluation of 
national policy at population, household and individual levels. The population covered by 
the Agincourt site is approximately eighty-thousand. 


For this test we us the physician-generated conditional prob a.bilities P an d initial guess of 
the CSMFs F' provided by Byass with the InterVA software (Byass, 2013). 


4.3 Results 

The results of our simulation study are summarized graphically as a set of Figures B-i 
The Agincourt results are presented in Figure]^ 

5 Discussion 

5.1 Summary of Findings 

InSilicoVA begins to solve most of the critical problems with InterVA. The results of ap¬ 
plying both methods to simulated data indicate that InSilicoVA performs well under all 
circumstances except ‘reporting errors’, but even in this situation InSilicoVA performs far 
better than InterVA. InSilicoVA and InterVA both perform relatively well when the simu¬ 
lated data are perfect. InSilicoVA’s performance is not affected by changing the magnitudes 
and ranges of the conditional probability inputs, whereas InterVA’s performance suffers dra¬ 
matically. With reporting errors both methods’ performance is negatively impacted, but 
InterVA becomes effectively useless. 

Applied to one specihc real dataset, both methods produce qualitatively similar results, but 
InSilicoVA is far more conservative and produces conhdence bounds, whereas InterVA does 
not. For Agincourt, Figure |^A shows the causes with the largest difference between the In¬ 
SilicoVA and InterVA estimates of the CSMF. InSilicoVA classihes a larger portion of deaths 
as due to causes labeled as ‘other.’ This indicates that these causes are related to either 
the communicable or non-communicable diseases, but there is not enough information to 
make a more specihc classihcation. This feature of InSilicoVA identihes cases that are diffi¬ 
cult to classify using available data and may, for example, be good candidates for physician 
review. 

We view this behavior as a strength of InSilicoVA because it is consistent with the fun¬ 
damental weakness of the VA approach, namely that both the information obtained from 
a VA interview and the expert knowledge and/or gold standard used to characterize the 
relationship between signs/symptoms and causes are inherently weak and incomplete, and 
consequently it is very difficult or impossible to make highly specihc cause assignments using 
VA. Given this, we do not want a method that is artihcially precise, i.e. forces hne-tuned 
classihcation when there is insufficient information. Hence we view InSilicoVA’s behavior 
as reasonable, ‘honest’ (in that it does not over interpret the data) and useful. ‘Useful’ in 
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the sense that it identihes where our information is particularly weak and therefore where 
we need to apply more effort either to data or to interpretation, like addition physician 
reviews. 


5.2 Future Work 


We plan a variety of additional work on InSilicoVA: 


1. Explore the possibility of replacing the Dirichlet distribution in (19), (23) and (27) 
with a mixture of Normals on the baseline logit transformed set of /„’s. This provides 
additional flexible parameters to allow each CSMF to have its own mean and variance. 


2. Embed InSilicoVA in a spatio-temporal model that allows F to vary smoothly through 
space and time. This would provide a parsimonious way of exploring spatio-temporal 
variation in the CSMFs while using the data as efficiently as possible. 


3. Create the ability to add physician cause assignments to (22) and (26) so that infor¬ 
mation in that form can be utilized when available. The physician codes will require 
pre-processing to remove physician-specihc bias in cause assignment, perhaps using a 
‘rater reliability method’ (for example: Salter-Townshend and Murphy| 2012). 


4. Most importantly, address the obviously invalid assumption that the signs/symptoms 
are independent given a specihc cause. This will require modeling of the signs/symptoms 
and the physician-provided conditional probabilities so that important dependencies 
can be accommodated. Further, this will require additional consultation with physi¬ 
cians and acquisition of new expert knowledge to characterize these dependencies. All 
of this will require a generous grant and the collaboration of a large number of experts. 
This will very likely greatly improve the performance and robustness of the method. 


5. Critically, re-elicit the conditional probabilities P from physicians so that they are 
logically well-behaved, i.e. fully consistent with one another and their complements. 

6. Focus and sharpen VA questionnaire. Quantify the influence of each sign/symptom to: 
(1) potentially eliminate low-value signs/symptoms and thereby make the VA interview 
more efficient, and/or (2) suggest sign/symptom ‘types’ that appear particularly useful, 
and potentially suggest augmenting VA interviews based on that information. 


7. Explore new possibilities for rehning the conditional probabilities P and potentially 
for entirely new models. 
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Figure 1: Simulation setup 1: ‘Fair Comparison’. A; InSilicoVA correctly assigns 
cause of death correctly effectively 100% of the time. InterVA is less accurate 
in assigning individual causes of death. B; InSilicoVA’s errors in identifying 
CSMFs are consistently very small. InterVA’s errors are also generally small, 
but the distribution has a long tail in the direction of large errors - sometimes 
InterVA’s errors are large. 
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Figure 2: Simulation setup 2: ‘Conditional Probabilities in the range [0.25 — 0.75]’. 

A; InSilicoVA correctly assigns cause of death correctly effectively 100% of the 
time. InterVA correctly assigns cause of death correctly 80% of the time with 
wide variation all the way down to as low as 60% and never above about 90%. B; 
InSilicoVA’s errors in identifying CSMFs are consistently very small. InterVA’s 
errors in identifying the CSMFs are larger and more variable. 
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Figure 3: Simulation setup 3: ‘Reporting Errors’. Both methods suffer, but InterVA 
suffers a lot more. A; InSilicoVA correctly assigns cause of death correctly about 
70% of the time. InterVA correctly assigns cause of death correctly about 40% 
of the time. B; InSilicoVA’s errors in identifying CSMFs are still consistently 
small. InterVA’s errors in identifying the CSMFs are larger. 
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Figure 4: Agincourt Application. A: Differences in CSMFs (InSilicoVA - InterVA) 
displaying the 10 specific causes that differ most. InSilicoVA is less willing to 
make highly specihc classihcations and thus produces larger CSMFs associated 
with less-specihc causes and smaller CSMFs associated with more specihc causes. 
B; Same as (A) but for the 10 largest differences in CSMFs that are still less than 
4%, which as a group includes HIV even though it is not in the top 10 that are 
plotted here. C: The CSMFs produced by both models on their natural scale, 
the same 10 causes as in A for which the differences are greatest. 
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